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Marked-Up Version of Substitute Specification 

Doscription 

Coding of image sequences with a pluralit>^ of image blocks and reference 
image s 

CODING OF IMAGE SEQUENCES WITH A PLURALITY OF IMAGE 
BLOCKS AND REFERENCE IMAGES 

BACKGROUND 

The invention relates to a method for coding a sequence of digitized images 
with a plurality of image blocks as well as a corresponding decoding method. The 
invention also relates to corresponding coding and decoding devices. 

Actual video coding standards (for example, see document [1] , below ) 
allow the coding of image sequences, in which cas e th e wherein macro image 
blocks used for an estimation of movement are updated by means of an intra-coding 
mode. As a resuh, errors are not reproduced in the image sequence. Updating by 
means of intra-coding modes can be carried out at regular intervals or based on the 
predetermined criteria. For aet^al -other video coding methods^ it is poGsibl e to use 
intra-coding modes^ that refer back to several previously coded reference images^ 
can be used . However, there are no mechanisms that allow an efficient video 
coding with inter-coding modes and intra-updating modes over error-prone 
networks. 

The publication "Proc. Intl. Conf On Image Processing ICIP, Lausanne, 
vol.1, 16.09.1996, pp.763-766 (Lio et al)" describes an intra-update method for 
video coding via channels prone to errors. This method analyses the specific 
sensitivity of macro blocks for channel errors and in this way obtains a 
specification for the intra-update modes. 
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The publication "Proc. IEEE ICASSP, San Francisco, vol. 5, 23.3.1992, pp. 
545-548 (Haskell et al)" describes several possible methods for resynchronizing 
movement-compensated videos that are adversely affected by ATM cell loss. 

Accordingly, a system and method is needed for allow an efficient video 
5 coding with inter-coding modes and intra-updating modes over error-prone 

networks . As will be discussed below, Therefor e , it is the obioct of th e inv e ntion to 
pr e par e _ a method is disclosed for coding a sequence of digitized images that uses a 
plurality of intra-coding and inter-coding modes as well as a plurality of reference 
images, and guarante e s t o ensure a reliable reconstraction of the digitized images in 
1 0 error-prone networks. 



SUMMARY OF THE INVENTION 

This object of th o inv e ntion is achiovod according to tho features of the 
independent patent claims. Further dev e lopments of the invention are produced by 
15 th e dependent claims. 

One exemplary embodiment Tho coding method according to tho inv e ntion 
codes a sequence of digitized images with a plurality of image blocks in error-prone 
networks , in which cas e wherein the macro blocks in a section of the image are 
coded in a first intra-coding mode depending on predetermined criteria. In 

20 addition F urthermore. the macro blocks in a section of the image are coded in a 

second intra-coding mode or in an inter-coding mode in which case wherein in the 
inter-coding mode for the macro blocks, movement vectors are selected firom the 
number of accessible reference images. The selection firom the number of 
accessible reference images is limited in such a way that referencing takes place 

25 from image areas that were not subjected to the first intra-coding mode at a later 
stage. This prevents h elps to prevent a r eference being made in the case of the 
inter-coding mode to the reference image areas that are subjected at least partially 
to an intra-coding mode. If the coding in the first intra-coding mode is carried out 
particulfirly for reasons of error robustness in order to avoid the reproduction of 

16 



Int'l Application No.: PCT/DE03/00176 

errors in the case of incorrect transmissions, this ensures that the coding is not 
based on image areas that were transmitted incorrectly. Therefore, an efficient and 
at the same time error robust coding, is guarant ee d p rovided in error-prone 
networks. 

5 Tn on e embodim e nt of the inv e ntio nU nder the exemplary embodiment 

discussed above , the coding is carried out in a first intra-coding mode at regular 
time intervals. Alternatively, the coding in the first intra-coding mode can be 
repeated at random time intervals. 

In a furthe r Under another exemplary embodiment, the coding is carried out 
10 in a second intra-coding mode or in an inter-coding mode for reasons of coding 
efficiency. For reasons of coding efficiency, an intra-coding mode is particularly 
taken into consideration if an object in the image sequence only appears 
temporarily in some images. 

Tn n prof e rr e d embodiment of tho inv e ntion Under yet another exemplary 

15 embodiment , the following steps are carried out to limit the reference images for 
coding a macro block. For each inter-coding mode from the number of possible 
inter-coding modes and for each reference image from the number of reference 
images that can be accessed by the rate distortion optimized movement 
compensation, optimized movement vectors are selected from the number of 

20 possible movement vectors. From a complete number that consists of the various 
possible combination of inter-coding modes and reference images, a limited 
number is created in which case the combinations that were coded in a later image 
in a first intra-coding mode are removed. Based on the limited nimiber and a 
number of intra-coding modes, the best combination based on rate distortion 

25 criteria is formed. For th e cas e , in which When the image block in the preceding 
aforementioned step was coded with an intra-coding mode, it is established in an 
additional step whether or not the image block was intra-coded on the basis of error 
robustness criteria (first intra-coding mode) or on the basis of the rate distortion 
optimization (second intra-coding mode). Therefore, an optimimi coding mode can 

30 be determined for macro blocks to be coded. tfeitt ^Utilization of r ate distortion 
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criteria can also be performed and is described in greater detail is alroady 
suffici e ntly loiown from th e prior art and, for e xample, d e Goribod in documents [3] 
and [4] below . 

Tn a proforr e d e mbodiment of the inv e ntion, th e Various rate distortion 
5 criteria are determined depending on the best combination of an error rate to be 
expected when transmitting the coded images. In this case, the distortion of the 
pixel values of the images is calculated particularly in order to determine these 
criteria. The distortion of the pixel values preferably contains the total of the 
quadratic differences between the pixel values before coding and the 
10 correspondingly decoded pixel values. Because the distortion is usually not known 
when coding, it is possible to estimate the distortion in a particularly preferred 
embodiment. 

In addition to the above-described coding method, tho inv e ntion relates to a 
corresponding method is also disclosed for decoding a sequence of digitized images 
15 in error-prone networks in which case the method is embodied in such a way that a 
sequence of digitized images coded with the coding method according to th e 
inv e ntio n described above is decoded. Under an example discussed in detail 
belo w in a pr e f e rr e d ombodimont of the decoding method , an error concealment is 
used for decoding. 

20 The invention also rolatos to oA device for coding a sequence of digitized 

images in error-prone networks is also disclosed, in which case the device is 
embodied in such a way that the coding method described above according to th e 
invention can be carried out. The invention also includes a corresponding device 
for decoding digitized images in error-prone networks in which case the device is 

25 embodied in such a way that the decoding method described above can be carried 
out. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The invention and its wide variety of potential embodiments will be more 
readily understood through the following detailed description, with referenc e to the 
accompanying drawing in which: 

Embodimonts of tho invention arc explained below on the basis of th e 
accompanying drawings. The drawings as follows 

Fig. 1 illustratest he- a section of a sequence of decoded images in which 

eas ewherein the images were previously coded with a method 
according to the prior art; and 

Fig. 2 illustrates a section of a sequence of decoded images in which 

eas ewherein the images were previously coded with the method 
according to the inventio n corr e sponding to Figiu-e 1 . 

DETAILED DESCRIPTION 

The image sequence shown in Figure 1 ¥r^as-is_coded with the encoder 
described in document [1] in which ease wherein this encoder carries out intra- 
updating modes in an intra-coding mode at regular intervals in order to avoid errors 
from being reproduced in the case of an incorrect transmission of the image 
sequence in the decoder. The intra-updating modes correspond to the coding 
modes in a first intra-coding mod e according to tho terminology of the claims . 

The image sequence is transmitted via the-an hitemet test pattern that is 
described in document [2]. In this case, the image sequence is transmitted in data 
packets in which oao e w herein a data packet consists of two rows of image blocks. 
In the text below, image blocks are referred to as macro image blocks whose 
shifting in the case of the inter-coding mode is determined by means of movement 
vectors. The coding method, by means of which the image sequence shown in 
Figure 1 was coded, also includes a second intra-coding mode and an inter-coding 
mod e in tho sonso of the terminology of tho claims . In the inter-coding mode, an 
estimation of movement with respect to a maximum of five reference image blocks 
is carried out. 
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The section of the image sequence shows the images in references N o. 9 to 
No. 12 (Fig. 1) of this sequence. In order to improve the display of the image 
sequence, a simpl ean error concealment by means of gray tones was also used. 
Under the example, W when the image sequence was transmitted, one packet was 
5 lost in the first image of the sequence. This transmission error is still displayed in 
image No. 9 of the sequence as can be seen by means of the horizontal lines in 
image No. 9 of Figure 1. _In image No. 10 of Fig. L an intra-updating mode of a 
section of the image blocks is carried out so that a section of the incorrect image 
area in image No. 10 has disappeared. In image No. 1 1 an inter-coding mode, by 

10 means of reference images, was carried out in which case the reference images lie 
temporally in front of image No. 10 and, therefore, do not include the intra- 
updating mode. Thus, a large part of the incorrect area again appears in image No. 
1 1 of Fig. 1 . The same phenomenon again appears in image No. 12. By means of 
this phenomenon, not only is the distortion in the image increased objectively, but 

15 the effect in the image is also found to be very disturbing firom a subjective 
viewpoint. 

The above-described image disturbances can be ascribed to the fact that for 
the coding used in the sequence of Figure 1, a first intra-coding mode is connected 
to an inter-coding mode that uses multiple reference images. The occurrence of 
20 these disturbances could be avoided by not returning to multiple reference images 
in the case of incorrect transmissions, but this would considerably reduce the 
compression efficiency. 

In order to avoid the above-described disturbances to the greatest possible 
extent, the coding method according to the invention limits the reference images to 

25 the effect that for the inter-coding mode only such reference image blocks are used 
that are not subjected to any intra-updating mode after the reference image has been 
coded. The results of the method according to tho invention are shown in Figure 2. 
Figure 2 she¥> ^illustrates t he same image sequence as in Figure 1 with the 
difference that the coding method according to the invention was used. From the 

30 images in Fig. 2, l it is clear that the image disturbances in images No. 1 1 and No. 
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12 have disappeared. ,This is due to the fact that for the inter-coding mode, no 
reference images that are transmitted incorrectly to the decoder are used. _The 
increase in the bit rate that resuUs from the method according to the invention is 
relatively moderate and Ues at ^tlv -about 5 %. 

5 Exemplary embodiments of the method according to the invention are 

described in greater detail below. _For an embodiment of the method for each 
macro block coding mode m is selected from the number of possible inter-coding 
modes Mp and for each reference image r from the number of accessible reference 
images R and optimum movement vectors v(m, r) from the number of movement 
10 vectors V(m) for the movement compensation. The selection takes place according 
to the rate distortion criteria. Mathematically, the rate distortion criteria are 
displayed as follows: 

v(m, r) = arg min(DDFD(m^ r, v) + A«otioiR««tio»(m, v)) (D 

ve V{m) 

in which case DoFoCm^ r, v) is the distortion according to the movement 
1 5 compensation and RmotionCm, r, v) contains the number of bits that are needed for 
coding the specific movement vector. _The fimction ((Ddfd(™, r, v) +XmotionRmotion 
(m, r, v)) is a so-called Lagrange cost fimction that contains the Lagrange multipUer 
\notion. .This fimction is minimized whereby optimum movement vectors are 
determined regarding the distortion and the memory space requirement for the 
20 movement vector. .Therefore, as a first result, optimized movement vectors v(m, r) 
are obtained for each reference image r and for each macro block coding mode m. 

In a next step, the number of movement vectors is limited by removing 
combinations from the nimiber consisting of the inter-coding modes Mp and the 
reference images R, in which referencing takes place from image areas that are 
25 subjected to an intra-updating mode at a later stage, for example, for reasons of 
error robustness. _Li this way, a nvmiber Op of possible values m and r is obtained 
for the movement vectors and this is as follows: 
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0. = {(w,r)e ^.,R}sr^M^AfM^^}' 

in which case 

k = 1, K is the number of an image block; 

f the vector {fi, fie} in which case the variable fi is the digit that gives the 
5 number of the reference image for the i-th image block for which the last intra- 
updating mode was carried out; and 

Sminfi(v(m,r),f,k) is an operation that determines the number of the reference image 
for the image block k depending on v(m,r) and f that, on the basis of the reference 
image limitation, is the last permitted reference image. 

10 Should I f the number of the last permitted reference image exceed the 

nimiber of the reference image r, then it consists of a combination (m, r) whose 
reference image is within the number of reference images limited by the method 
according to the invention. Should If the last permitted reference image be less 
than the reference image r, then the corresponding combination (m, r) will be 

15 rejected. 

The Umited set Op of reference images and inter-coding modes m resulting 
from the previous step is combined with a set of intra-coding modes Mi that can be 
used in the method according to the invention and the optimized coding mode 0(k) 
is again determined from the set union 0 = {Mi, Op} for each macro block k by 
20 means of the rate distortion criteria. _If this macro block is intra-coded of necessity, 
for example, by regular or random intra-updating modes then the number of 0 is 
limited to only intra-modes, that is 0 = Mj. _Of course, in this case it is also not 
necessary to determine Op. .Mathematically, the rate distortion criteria can again be 
formulated as the minimizing problem of a Lagrange cost fiinction: 

o{k) = arg mm {D{o) + Am^{o)) , ( 3 ) 

25 
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in which case R(o), describes the number of bits, codes the image block in the 
coding mode o, and D(o) represents the distortion for this coding mode. 

If in the m e thod according to tho invention there is a regular or random 
intra-updating mode in the example above , the distortion is produced as the sum of 
5 the quadratic differences betw^een the original image block and the image block 

received after the decoding.^ If the intra-updating mode should be carried out on the 
basis of an error-optimized channel adaptive coding described further below, the 
distortion is given in the decoder as the expected value of the distortion. 

In a nex ^following step, it is still necessary to establish whether or not an 
10 intra-coded image block was intra-coded because of error robustness reasons in 
order to avoid the reproduction of errors or for reasons of coding efficiency. _An 
intra-coding mode for reasons of coding efficiency particularly prevails if an object 
in the image sequence only appears temporarily.. For an intra-coding mode because 
of coding efficiency reasons, a reference image limitation is not desired. _In order to 
1 5 determine the reasons for the intra-coding mode, a rate distortion optimization is 

again performed according to equation (3), but where the total number 0 = {Mj, Op} 
is always u sed and as the distortion measurement the total of the quadratic 
differences between the original image block and the image block received after 
decoding. _The result of the optimization is designated as 6(k). Subsequently, an 
20 error robustness flag Ck is set in which case Ck = 5o(k)^(k) and Scondition is the 

Kronecker symbol that is 1 if the condition has been met and otherwise has the 
value 0. .Therefore, the intra-coding mode was carried out for reasons of error 
robustness if the flag is set to 1 . 

If all the image blocks of an image were processed, the vector f is updated 
25 for all the entries fk for which the error robustness flag Ck is set at 1 . _As a result, a 
reference image limitation is avoided for such intra-coding modes that were 
performed for reasons of coding efficiency and thus the appearance and 
disappearance of objects can be efficiently executed by means of coding with the 
aid of a number of reference images. 
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An e mbodim e nt of th e mothod according to the invention is A nother 
exemplary embodiment is described below for which a channel-adaptive reference 
image is selected on the basis of rate distortion criteria. .For thi s example, the 
distortion D(o) has to be estimated in the decoder. .Possibilities of estimating this 
5 distortion ar e Icnown, for e xampl e can be determined by using any of the methods 
described inr ^em documents [5], [6] and [7] below . ^ ^Another p ossibility of 
determining the distortion is the incorporation of the random channel behavior C 
when the distortion is estimated, .After an image n has been transmitted, the 
channel behavior C is in this case given by means of the binary sequence {0, 1 }^^"^ 

10 in which case p(n) is the number of packets to be transmitted that are needed to 
transmit the images 1 to n. _In this case, a 0 in the sequence designates a correctly 
received packet whereas a 1 indicates a lost packet. _The random variable that 
describes the binary sequence up to image n is designated as Cp(n). ^The pixel 
distortion in the decoder d e pends w ill depend on the pixel value reconstructed in 

15 the decoder that is designated as Si and which is unknown to the encoder carrying 
out the coding. .The pixel distortion depends on the channel behavior C and the 
selected coding mode o, i.e: Si = Sj (Cp(n),o). .The distortion is estimated as the total 
of all the expected values of the quadratic pixel distortions di(o) of all the macro 
blocks i in which case it is assumed that the channel behavior Cp(n) is known to the 

20 encoder. .The pixel distortion di(o) for the macro block i is as follows: 

in which case Ecp(n-i) represents the expected value of the quadratic difference of 
the original pixel value and the reconstructed pixel value averaged over the channel 

Cp(n.l). 

25 In order to calculate the expected value, the following eas^method can be 

used. .It is assumed that T copies of the random variables "channel behavior" are 
available in the encoder, .These copies are designated as Cp(n)(t), with t = 1, T. .It 
is also assumed that all the random variables Cp(n)(t) are distributed independently, 
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identically and statistically. .Therefore, according to the strict law for high 
numbers, T -> oois as follows: 



- 5*(C(.)(r),o)j^ = £c,c-)5.-5i(Ci-)>oj2 = d{o). 



(5) 



Therefore, with the expression on the left side, the expected value di(o) can 
5 be estimated and in a next step, the expected distortion Di(o) calculated.. The 

reconstruction of the pixel values depends on the channel behavior Cp(n-i)(t) as well 
as the concealment in the decoder.. By means of the last-mentioned formula it is 
possible to estimate in the encoder the intensity of the distortion in the decoder. 

In addition, although the invention is described in connection with digitized 
10 images, it should be readily apparent that the invention may be practiced with any 
type of still or moving digital image format. It is also understood that the process 
portions and segments described in the embodiments above can substituted with 
equivalent processes to perform the disclosed methods and processes. Accordingly, 
the invention is not limited by the foregoing description or drawings, but is only 
15 limited by the scope of the appended claims. 
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AbGtraot 

Method for coding a sequence of digitized images with a plurolity of iiriflg e 
blocks 

ABSTRACT 

¥he-A coding method according to the invention codes a sequence of 
digitized images with a plurality of macro blocks in error-prone networks in which 
case the macro blocks in a section of the image are coded in a first intra-coding 
mode depending on predetermined criteria. ^In addition, the macro blocks in a 
section of the image are coded in a second intra-coding mode or in an inter-coding 
mode in which case in the inter-coding mode for the macro blocks, movement 
vectors are selected from the number of accessible reference images. The selection 
fi-om the number of accessible reference images is limited in such a way that 
referencing takes place from image areas that were not subjected to the first intra- 
coding mode at a later stage. 
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REMARKS 

The present amendment makes editorial changes and corrects typographical 
errors in the specification, which includes the Abstract, in order to conform the 
specification to the requirements of United States patent practice. No new matter is 
5 added. Licluded is a Substitute Specification including a marked-up version of the 
changes made via by the present amendment. 

In addition, the present amendment cancels original claims 1-13 in favor of 
new claims 14-23. Claims 14-23 have been presented because revisions by red- 
lining and underHning which would have been necessary in claims 1-13 in order to 
10 present this claim in accordance with preferred United States patent practice would 
have been too extensive, and, thus, too burdensome. The present amendment is 
intended for clarification purposes only and not for substantial reasons related to 
patentability pursuant to U.S.C. §§101, 102, 103 or 1 12. Indeed, the cancellation of 
claims 1-13 does not constitute an intent on the part of the Applicants to surrender 
1 5 any of the subj ect matter of the claims. 

Finally, the Intemational Search Report is included in the record 
accompanying the Intemational Patent Document WO 03/060877. 

Early consideration on the merits is respectfully requested. 

Respectfully submitted, 

20 
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